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METHOD AND APPARATUS FOR CONTROLLING PIXEL SENSOR 

ELEMENTS 

5 Claim of Priority 

This patent application claims the benefit of U.S. Provisional Application No. 
60/1495796 filed August 19, 1999, which is hereby incorporated herein by reference in its 
entirety. 

Background of the Invention 

10 Field of the Invention 

The present invention relates to imaging systems, and in particular, to methods 
and systems for controlling pixel sensor elements. 
Description of the Related Art 

Conventional integrated circuit imaging devices include an array of light 

15 detecting elements or pixels which are interconnected to generate a signal representation 

of an image illuminating the device. Two common examples of conventional integrated 
circuit imaging devices are a charge coupled device (CCD) and a complementary metal 
oxide semiconductor (CMOS) image sensing device. Conventional imaging devices use 
one or more light detecting elements and charge storage elements. In order to produce a 

20 color image, the imaging devices separate the light into various color components by 

filtering the light before the light strikes the fight detecting elements. The array of light 
detecting elements is often deposited with a filter layer such that neighboring pixels 
may have different color filters and organized in a particular pattem. 

Because each pixel is only capable of detecting a single color, conventional 

25 imaging devices require a process by which all of the color components are 

reconstructed for each pixel in order to maintain the original unfiltered array resolution. 
To reconstruct the color components, conventional imaging devices use a process of 
color interpolation that is performed after an analog signal associated with each pixel 
has been digitized. The conventional process of color interpolation performed after an 

30 analog signal associated with each pixel has been digitized requires conversion fi*om 

analog to digital (A/D) and may require extensive computations in order to achieve a 
high quality color presentation of the image. The A/D conversion and extensive 



computations may require hardware, such as analog-to-digital (A/D) converters, 
memory, processors and software. The hardware and software may add to the 
complexity, size and expense of the imaging device and reduce the speed of the imaging 
process. 

5 Conventional imaging devices also require color compensation for differences 

in the response of the various color filters and for variations within the integrated circuit 
sensor array, such as process, materials, temperature or manufacturing. For example, 
when the primary color scheme is used, the response of an element that absorbs red 
light may be different than an element that absorbs blue light even when illuminated by 

10 light of equal red and blue luminosity levels. When exposed to a flat light image 

having equal intensity and chromatisity levels throughout, typical CMOS or CCD 
arrays may generate analog signals having significant magnitude variations for the 
different color components. Accordingly, if the analog signals are used to reproduce 
the original image, the reproduced image colors will not match the original colors. 

15 To overcome this problem, conventional imaging devices employ a process of 

color correction that is performed after the analog signals for each pixel have been 
digitized. One drawback of the conventional color correction process is a loss in color 
dynamic range that results from under-utilization of the A/D converter for some of the 
color components. Another drawback is increased computations that translates into 

20 additional hardware, size, expense and/or reduced speed. 

The effect of loss in color dynamic range is particularly noticeable in the lov/ 
light areas of an image that contains low as well as high light regions. The human eye 
is sensitive to minute changes in hue and saturation levels. A reduction in color 
component dynamic range may result in less vivid, plain, or flat images. Attempts to 

25 correct this via hue or saturation enhancement filters may cause color distortion rather 

than color restoration. 

Another color compensation that integrated circuit color imaging devices 
require is for different illumination temperatures. The hue of a color component 
changes with respect to the ambient illumination. Thus, a white object under sunlight 

30 conditions is perceived by the imaging device as white, but under fluorescent light 

conditions is perceived as hght green. 
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To overcome this problem, conventional integrated circuit color imaging 
devices employ a process of white balance that is performed after the analog signals for 
each pixel have been digitized. Typically, each of the three processed color 
components, red, green or blue, is a convolution of the three colors. A drawback of the 
5 conventional white balance process is increased computations that translates into 

additional hardware and/or reduced speed. 

The collection of signals read from the pixels represents the image viewed by 
the array. Each pixel represents a sample of the image and hence is a data value in the 
two-dimension image produced by the imaging system. Defect pixels, referred to as 

10 'bad pixels/ do not contain a correct value and appear as artifacts. The bad pixels can 
reduce the image quality significantly. A bad pixel is caused by array defect and 
produces an output signal that significantly deviates from the mean output level of 
adjacent pixels when the exposure level of all pixels is substantially unified. Pixels that 
are significantly brighter than adjacent pixels in a unified dark frame are referred to as 

15 'hot pixels,' while pixels that are significantly darker than adjacent pixels in a unified 

bright frame are referred to as 'dead pixels.' 

The defect pixels are typically distributed in a random manner. However, a bad 
column (i.e. - a complete column is defective), or a blemish (i.e. - a cluster of 
neighboring pixels is defective) may occur and are typically discarded by the 

20 manufacturer. The manufacturer releases sensor arrays that contain random defect 
pixels in an amount that does not exceed a given limit, and the bad pixels are typically 
corrected. Both CCD and CMOS integrated circuit color imaging devices employ a 
process of bad pixel detection and correction. Conventionally, the detection step is 
performed off-line by the manufacturer. A bad pixel Ust is stored in the device. The 

25 correction step is typically performed after the analog signal for each pixel has been 

digitized. 

Summary of the Invention 
The present invention is directed to imaging methods and systems for flexibly 
30 addressing and processing imaging pixel sensor elements. The novel architecture of the 
present invention allows for a highly integrated, low cost imager with high speed 
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performance and good image quality. For example, the imaging system provides on- 
the-fly color interpolation, color compensation (also called color correction, color 
maximization or white balance) and/or fixed pattern noise reduction. The hardware 
and/or software related to on-the-fly color interpolation, color compensation and/or 
5 fixed pattem noise reduction may be provided on-chip. 

One embodiment of an imaging system in accordance with the present invention 
comprises an array of pixel sensor cells arranged in rows and columns, a plurahty of 
detection circuits, an array controller, a control circuit that allows for various 
programmable modes of pixel readout, and a programmable amplification stage that 

10 may be adjusted per pixel or per group of pixels. In accordance with one embodiment, 

the imaging system further includes a color filter layer deposited on the sensor array. 
The color filter may comprise, for example, Red, Green, and Blue filters organized in 
the Bayer pattem scheme. Li one embodiment, the system further contains a circuit for 
performing one or more averaging operations. 

15 One aspect of the present invention relates to an on-the-fly color interpolation 

apparatus and process that comprises a color reconstruction procedure performed during 
the sensor readout stage. The on-the-fly color interpolation process is capable of 
reading two or more consecutive rows and two or more consecutive columns 
simultaneously, summing some of the signals, and independently ampUfying the signals 

20 with an optionally programmable gain ampUfier. The on-the-fly color interpolation 

process is advantageously performed in an analog domain before an analog-to-digital 
conversion. 

The on-the-fly color interpolation process of the present invention provides higli 
speed of operation and reduced computational complexity with good color image 

25 quality. The on-the-fly color interpolation process is advantageously suited for many 

imaging applications today that do not require high color precision or high image 
quality, as well as those which do. For example, some high quality, digital cameras 
have modes of operation such as a 'preview* mode where image quahty is less important 
than speed and implementation complexity. 

30 One embodiment of the on-the-fly color interpolation process provides at least 

two modes of operation: a full resolution mode and a sub-sampling mode. Full 
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resolution mode is preferably used when high image quality is desired. In full 
resolution mode, each Red-Green-Blue (RGB) triplet needed for a color pixel 
representation is produced from a group of 2x2 pixels within the Bayer pattern. The 
four pixels are read out simultaneously, the two diagonally neighboring Green pixels 
5 are summed together, and the gain associated with the Green pixels is reduced in half. 

The resulting Green component is sent out with the diagonally neighboring Red and 
Blue pixels to produce one RGB triplet. Each subsequent RGB triplet shares one of the 
Green pixels and either the Red or Blue pixel of the preceding RGB triplet. 

The sub-sampHng mode may be used when a lower resolution image is desired, 

10 such as when a user previews an image. Sub-sampling is achieved via skipping pixels 

along the horizontal and/or the vertical axis of the pixel array. For a sub-samphng ratio 
of l:j vertically and l:k horizontally, where j, k are even, each RGB triplet that is 
needed for a color pixel representation is produced from a group of j by k pixels with 
the Bayer pattern. All Green pixels in the j*k neighborhood are averaged to produce 

15 the green component. Similarly, all of the Red pixels in the same neighborhood are 

averaged to produce the Red component, and all Blue pixels in the neighborhood are 
averaged to produce the Blue component. The three components are then sent out as 
the RGB triplet representing the j by k group of pixels. The sub-samphng process can 
be achieved by reading the j rows simultaneously. 

20 Both the full resolution and the sub-sampling modes of operation can be fiirther 

combined with a window readout mode where only a sub-region of the whole sensor 
array is read. 

Another aspect of the present invention relates to an apparatus and process for 
color compensation or maximization. The color compensation process compensates for 

25 differences in the response of various color filters and variations within the integrated 

circuit sensor array, such as process, materials, temperature or manufacturing. The 
color compensation process of the present invention improves color dynamic range, 
decreases computations and hardware and improves speed of operation. The color 
compensation process of the present invention is advantageously performed in the 

30 analog domain before an analog-to-digital conversion. The color compensation process 
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of the present invention also improves white balance which is used to compensate 
colors for different illumination temperatures. 

In one embodiment, the imaging system with color compensation includes a 
control circuit that provides four output paths that can be amplified separately via one or 
5 more stages of programmable gain ampUfiers and/or summing amplifiers. The control 

circuit allows four independent readouts for even rows, odd rows, even columns and 
odd columns, thus providing separate and/or simultaneous output paths for Red, Green 
and Blue pixels. 

For example, one embodiment of the imaging system employs a readout control 
10 that provides outputs for the Red pixel, the Blue pixel, and the two Green pixels. Thus, 

the imaging system allows further gain compensation for a Green pixel that resides in an 
even column compared to a Green pixel that resides in an odd column. The four signals 
are then ampHfied via four corresponding programmable gain amplifiers. The readout 
control logic ensures that the row and column switches are closed in the appropriate 
15 sequence. 

The present invention does not require the integrated color imaging system to 
employ a simultaneous readout of the n-by-n pixel block. A 'pipeline' approach may be 
utilized instead of a parallel readout. The pipehne approach uses one or more analog 
line storage units, e.g., capacitors. For example, in one embodiment using the Bayer 

20 color pattem, two line storage units are used. The first of two consecutive lines that is 

readout from the array is stored in the furst line storage unit. The second line is 
averaged with the stored line to produce the RGB triplets, while a "first" line of the next 
two consecutive lines is readout and stored in the second line storage unit and so on. 
Thus, the two line storage units are used in a 'ping pong' fashion. 

25 The imaging system does not restrict the type of transfer function that is 

implemented in the programmable gain ampUfiers. Each of the four amplifiers can 
implement a different transfer function such as log or an exponent where the power 
value is programmable. Thus, each color can be optimized independently for maximum 
dynamic range. 

30 In one embodiment of the imaging system with color compensation, the imaging 

system further employs several gain stages for color convolutions associated with white 
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balance. The imaging system employs a readout control that provides three outputs for 
the Red pixel, the Blue pixel, and either the even Green pixel or the odd Green pixel via 
a multiplexor. The three output signals are then amplified via nine programmable gain 
amphfiers and summed via three summing ampUfiers accordingly. The readout control 
5 logic ensures that the row and column switches are closed in the appropriate sequence. 

Another aspect of the invention relates to a fixed pattern noise reduction 
apparatus and process. The fixed pattern noise reduction process reduces noise related 
to pixel-to-pixel variation. This variation is primarily due to dark current leakage, 
which may be integrated together with the signal and hence contaminate the signal. The 
10 dark current leakage may be due to thermal generation in the neutral bulk material, in 

the depletion region and due to surface states. The dark current level may vary betweeti 
pixels and may be particularly noticeable between columns due to column buffers. 

The fixed pattern noise reduction process of the present invention allows 
increased dynamic range (high image quality), high speed of operation and reduced 
15 computational complexity. The fixed pattern noise reduction of the present invention is 

advantageously performed in the analog domain before an analog-to-digital conversion. 

In the imaging system of the present invention with fixed pattern noise 
reduction, the array of pixels comprises a group of exposed pixels and a group of dark 
pixels. In one embodiment, the dark pixels are deposited with an opaque mask layer 
20 and thus are not exposed to light. In one embodiment, the programmable readout 

control circuit has a programmable non-destructive readout mode. 

In one embodiment, the array includes a row of dark pixels. Each image pixel 
value is produced from a combination of an exposed pixel and a dark pixel that resides 
in the same column. The two pixels are read simultaneously, and the dark value is 
25 subtracted from the exposed value. In another embodiment, the array includes several 

rows of dark pixels. 

In one embodiment, the imaging system produces a black and white image. In 
another embodiment, the imaging system further comprises a color filter layer 
deposited on the exposed pixel sensor elements. In one embodiment, the color filter 
30 layer comprises Red, Green, and Blue filters organized in the Bayer pattern. In another 

embodiment, the color filter layer comprises Yellow, Cyan and Magenta filters. 



-7- 



In other embodiments, other color filter systems and/or other patterns or 
configurations may be used. In addition, other embodiments of the imaging system do 
not have pixels organized in a rectangular matrix. 

In one embodiment, at least a portion of the sensor cells are active. In another 
5 embodiment, at least a portion of the sensor cells are passive. 

The imaging system in accordance with the present invention may also include 
additional on-chip or off-chip amplification stages, analog-to-digital conversion units, 
memory units and various other signal processing blocks. In one embodiment, the 
imaging system further comprises a micro-lenses layer. 

10 In one embodiment, the imaging system further contains a control circuit that 

allows for special pixel readout modes and a circuit for performing an averaging and/or 
a subtraction operation. 

In one embodiment, the imaging system employs an on-the-fly fixed pattem 
noise reduction process that subtracts dark current during the sensor readout stage. The 

15 on-the-fly fixed pattem noise reduction process is capable of reading two consecutive 

rows simultaneously by a readout shift register and a row readout control, re-reading a 
dark row together with an exposed row, subtracting a dark row value fi'om an exposed 
row value with a summing amplifier and amplifying the difference with a 
programmable gain amplifier. 

20 In another embodiment, the imaging system employs a mode of on-the-fly fixed 

pattem noise reduction that subtracts a dark current average value during the sensor 
readout stage. The imaging system is capable of reading three consecutive rows 
simultaneously by a column shift register and a row readout control, re-reading the dark 
rows together with an exposed row, averaging the dark rows with a summing amplifier, 

25 subtracting the averaged dark row value from the exposed row value with a summing 

amphfier, and amplifying the difference with a programmable gain amphfier. The 
imaging system utiUzes several dark rows for improved quality. Each image pixel 
value is produced from a combination of a current exposed pixel and a dark current 
value that is the average of two dark pixels residing in the same column. 

30 The present invention does not limit the number of dark rows that are averaged 

nor does it restrict the readout mode options. Furthermore, in other embodiments of the 
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present invention, the on-the-fly fixed pattern reduction can be performed with or 
without on-the-fly color interpolation, with or without sub-sampling, and can be further 
combined with a window readout mode where only a sub-region of the whole sensor 
array is utilized. 

5 Another aspect of the present invention relates to an integrated circuit imaging 

system that offers on-line bad pixel correction process that can be performed in the 
analog domain and thus provides high speed operation. 

Brief Description of the Drawings 
10 Figure 1 illustrates one embodiment of an imaging system coupled to a 

television. 

Figure 2 illustrates one embodiment of an imaging system coupled to a 
computer. 

Figure 3 illustrates a Bayer pattern color filter for the primary color system. 
15 Figure 4 illustrates one embodiment of a CMOS integrated circuit color imaging 

system that supports on-the-fly color interpolation. 

Figure 5 illustrates one embodiment of a readout sequence for the system of 
Figure 4 with on-the-fly color interpolation in a full resolution mode. 

Figure 6 illustrates one embodiment of the on-the-fly color interpolation process 
20 in a full resolution mode. 

Figure 7 illustrates one embodiment of a readout sequence for on-the-fly color 
interpolation in a sub-sampling mode. 

Figure 8 illustrates one embodiment of a CMOS integrated circuit color imaging 
system that supports on-the-fly color interpolation in a sub-sampling mode. 
25 Figure 9 illustrates one embodiment of the on-the-fly color interpolation process 

in a sub-sampling mode. 

Figure 10 illustrates a one embodiment of window mode. 
Figure 1 1 illustrates one embodiment of a readout control that supports on-the- 
fly color interpolation in a full resolution mode as shown in Figure 5. 
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Figure 12 illustrates one embodiment of a readout control that supports on-the- 
fly color interpolation in sub-sampling mode shown in Figure 8, as well as the full 
resolution mode shown in Figure 5. 

Figure 13 illustrates one embodiment of a CMOS integrated circuit color 
5 imaging system with color correction. 

Figure 14 provides one embodiment of a process for color correction. 

Figure 15 illustrates one embodiment of a CMOS integrated circuit color 
imaging system with white balance. 

Figures 16, 16A and 16B provide one embodiment of a process for white 
10 balance. 

Figure 17 presents embodiments of optional transfer function for the 
programmable gain ampUfiers. 

Figures 18, 18A and 18B provide an another embodiment of the process shown 
in Figure 16. 

15 Figure 19 illustrates another embodiment of a CMOS integrated circuit color 

imaging system with white balance. 

Figure 20 illustrates one embodiment of an imaging system that supports fixed 
pattern noise reduction. 

Figure 21 illustrates one embodiment of a fixed pattem noise reduction process. 
20 Figure 22 illustrates another embodiment of an imaging system that supports 

fixed pattem noise reduction. 

Figure 23 illustrates another embodiment of a fixed pattem noise reduction 
process. 

Figure 24 illustrates one embodiment of a CMOS integrated circuit imaging 
25 device implementation that accommodates on-the-fly bad pixel correction. 

Figure 25 illustrates a flowchart for the on-the-fly bad pixel correction of Figure 

24. 

Detailed Description of the Preferred Embodiments 
30 The present invention relates to a novel imaging system that provides flexible 

addressing and processing of imaging pixel sensor elements. The novel architecture of 
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the present invention allows for a highly integrated, low cost imager with high speed 
performance and good image quality. For example, the imaging system may provide 
on-the-fly color interpolation, color compensation (also called color correction, color 
maximization or white balance) and/or fixed pattern noise reduction. 
5 The exemplifying imaging systems described below with reference to Figures 1- 

23 use a CMOS integrated circuit, an array of pixels organized in a rectangle matrix, 
and a color filter with a primary color system (RGB) in a Bayer color pattern. The 
imaging systems of the present invention may be implemented with a charge coupled 
device (CCD) or other imaging technologies. Likewise, the imaging systems of the 

10 present invention may be implemented with another color system, such as the 
comphmentary color system (Yellow, Cyan and Magenta) and/or another color pattern. 
In addition, the imaging system of the present invention may be implemented with the 
pixels organized in another pixel matrix or pixel topography. For example, the array 
does not have to be rectangular. 

15 The imaging system may also include either on-chip, as is possible with CMOS 

integrated circuit imaging devices, or off-chip, as is the case with CCD integrated 
circuit imaging devices, amplification stages, analog to digital conversion units, 
memory units and various other signal processing blocks. In addition, the system may 
further comprise a micro-lenses layer. 

20 In one embodiment, the color interpolation system, the color compensation 

system and/or the fixed pattern noise reduction system reside together with the sensor 
array in the same chip, such as in a CMOS integrated circuit color imaging device. In 
another embodiment, the color interpolation system, the color compensation system 
and/or the fixed pattern noise reduction system reside in a separate companion chip, 

25 such as in a CCD integrated circuit imaging device. 

Figures 1 and 2 illustrate exemplifying systems incorporating the novel imaging 
system 100, 102. The imaging systems 100, 102 will be described in greater detail 
below with reference to Figures 3-23. 

Figure 1 illustrates one embodiment of an imaging system 100 coupled to a 

30 television 116 via a coax cable 114. The exemplifying system 100 includes a lens 102, 

a sensor array 104 which may have color filters, a readout control 106, gain amplifiers 
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108 for each color, a NTSC encoder 110 including gamma correction and a power 
supply 112. In one embodiment, the imaging system 100 is a video camera. In other 
embodiments, the imaging system 100 may be implemented in security cameras, digital 
cameras, camcorders, video telephones and the like. 
5 Figure 2 illustrates one embodiment of an imaging system 122 coupled to a 

computer 126 via a USB cable 124. The exemplifying system 122 includes a lens 102, 
a sensor array 104 which may have color filters, a readout control 106, gain amplifiers 
108 for each color, an analog to digital converter 1 18, a USB interface 120 and a power 
supply 112. In one embodiment, the imaging system 122 is a video camera. In other 

10 embodiments, the imaging system 122 may be implemented in security cameras, digital 
cameras, camcorders, video telephones and the like. 

Figure 3 illustrates a conventional Red, Green and Blue (primary color system) 
Bayer color pattern 130 for a color filter that is deposited on an array of pixel cells that 
detect hght. The pattern core is a group of 2 by 2 pixels that contains 2 green 

15 components 134, 136, one red component 132 and one blue component 138. The 2 

green components 134, 136 are diagonal neighbors, and the red and the blue 
components 132, 138 are diagonal neighbors. Thus, the green resolution of the array is 
reduced by a ratio of 2:1 horizontally only, while the red and the blue resolution is 
reduced by a ratio of 2:1 horizontally and vertically. The pattern 130 exploits the fact 

20 that the human eye perceives intensity edges better than color edges and that the green 
component contains the highest amount of intensity information. 
On-The-Flv Color Interpolation 

As will now be described, an imaging system with a flexible pixel address 
scheme allows for on-the-fly interpolation of colors based on the outputs of two or 

25 more pixels. On-the-fly color interpolation relates to color reconstruction during the 

sensor readout stage. Color reconstruction relates to reconstructing desired color 
components for each pixel in order to maintain the original unfiltered array resolution 
and to compensate for the fact that each filtered pixel is only capable of detecting a 
single color. 

30 The on-the-fly color interpolation process of the present invention provides high 

speed of operation and reduced computational complexity with good color image 
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quality. The on-the-fly color interpolation process is advantageously suited for many 
imaging applications that do not require high color precision or high image quality, as 
well as those which do. For example, some high quality, digital cameras have modes of 
operation such as a 'preview' mode where image quality is less important than speed 
5 and implementation complexity. 

One embodiment of the on-the-fly color interpolation process provides at least 
two modes of operation: a full resolution mode and a sub-sampling mode. Full 
resolution mode is preferably used when high image quality is desired. In full 
resolution mode, each Red-Green-Blue (RGB) triplet needed for a color pixel 

10 representation is produced from a group of 2x2 pixels within the Bayer pattern. The 
four pixels are read out simultaneously, the two diagonally neighboring Green pixels 
are summed together, and the gain associated with the Green pixels is reduced in half 
The resulting Green component is sent out with the diagonally neighboring Red and 
Blue pixels to produce one RGB triplet. Each subsequent RGB triplet shares one of the 

15 Green pixels and either the Red or Blue pixel of the preceding RGB triplet. 

Sub-sampling mode is preferably used when a lower resolution image is desired, 
such as a preview feature. Sub-sampling is achieved via skipping pixels along the 
horizontal and/or the vertical axis of the pixel array. For a sub-sampling ratio of l:j 
vertically and 1 :k horizontally, where j, k are even, each RGB triplet that is needed for a 

20 color pixel representation is produced from a group of j by k pixels with the Bayer 

pattern. All Green pixels in the j*k neighborhood are averaged to produce the green 
component. Similarly, all of the Red pixels in the same neighborhood are averaged to 
produce the Red component, and all Blue pixels in the neighborhood are averaged to 
produce the Blue component. The three components are then sent out as the RGB 

25 triplet representing the j by k group of pixels. The sub-sampling process can be 

achieved by reading the j rows in parallel or in series. 

Both the ftiU resolution and the sub-sampling modes of operation can be fiirther 
combined with a window readout mode (also called "window mode" or "windowing"). 
In window mode, a sub-region of the whole pixel sensor array is readout and processed. 

30 Thus, a window mode is essentially a cropping operation that produces a smaller area of 

interest. Window readout mode is faster and provides a higher frame rate. Windov/ 
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mode may be used in digital cameras for exposure and focus calculations, for electronic 
zoom and more. Window mode is described in greater detail below with reference to 
Figure 10. 

Figure 4 illustrates one embodiment of a novel CMOS integrated circuit color 
5 imaging system 140 that supports on-the-fly color interpolation using analog signals. 
The system 140 uses the Bay color pattern illustrated in Figure 3. As illustrated in 
Figure 4, the system 140 includes a column readout control circuit 146, a Hne 
synchronization signal 142, a pixel clock 144, a first column readout line 148, a second 
column readout line 148', a third column readout line 148", a first switch 150, a first 

10 column buffer 152, a second switch 150', a second column buffer 152', an analog 
summing amplifier 154, a programmable analog red gain amplifier 156, a red video out 
line 158, a red gain control line 160, a green video out line 162, a green gain control 
line 164, a programmable analog green gain ampUfier 166, a blue video out line 168, a 
blue gain control line 170, a programmable analog blue gain amplifier 172, a red output 

15 path 174, a green output path 176, a blue output path 178, a row readout control circuit 

180, a frame synchronization line 182, a line synchronization line 184, a first row 
readout line 186, a second row readout line 186', a first row buffer 188, a second row 
buffer 188', a first red pixel 190, a first green pixel 192, a second green pixel 194 and a 
first blue pixel 196. 

20 In the description herein, a "programmable" component refers to a component 

that responds to a command from an end-user of the imaging system or to a command 
issued by internal firmware according to firmware stored in the imaging system. For 
example, if an end-user chooses a 'zoom' fimction or a 'preview' fimction on a video 
camera containing the imaging system of the present invention, the imaging system 

25 directs the programmable components to act in a predefined manner according to 

firmware stored in the imaging system. 

The imaging system of the present invention does not need the color gain 
amplifiers 156, 166, 172 (Figure 4) to be implemented as a separate stage. In one 
embodiment, a plurality of color gain amplifiers are contained within the pixel circuitry 

30 of tiie sensor array. In another embodiment, a plurality of color gain amphfiers are 
contained within the column buffers 152, 152'. 
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As will now be described, the system 140 accommodates on-the-fly color 
interpolation via a programmable pixel readout mode. The exemplifying system 140 
implements an RGB Bayer color pattern on-the-fly color interpolation when windowing 
and sub-sampUng are not active. The system 140 is not limited in its mode of operation 
5 and can support windowing and sub-sampling via the programmable readout control 

circuitry 146, 180, the summing amphfier(s) 154 and the programmable gain amplifiers 
156, 166, 172. 

Figure 5 illustrates one embodiment of a pixel readout sequence for the system 
140 of Figure 4 with on-the-fly color interpolation in a full resolution mode. The 
10 sequence accommodates an RGB Bayer pattern color filter 130 (Figure 3) in a non- 

window mode and without sub-sampling. Each RGB triplet that is needed for a color 
pixel representation is produced fi-om a group of 2x2 pixels that contains the Bayer 
pattern core. For example, while reading four pixels substantially simultaneously, the 
two diagonally neighboring Green pixels 192, 194 are averaged and sent out with the 
15 neighboring Red and Blue pixels 190, 196 to produce a RGB triplet. Three other 

subsequent RGB triplet shares one of the Green pixels 192, 194 and either the Red or 
the Blue pixels 190, 196 with the previous RGB triplet. For example, in Figure 5, four 
RGB triplets are formed from a 3x3 pixel block 200: 

{R(0,0), [(G(1,0) + G(0,l))/2], B(l,l)} 
20 {R(2,0), [(G(1,0) + G(2,l))/2], B(l,l)} 

{R(0,2), [(G(0,1) + G(l,2))/2], B(l,l)} 

{R(2,2), [(G(2,l) + G(l,2))/2], B(l,l)} 
where the first numeral in the parenthesis represents the row and the second numeral 
represents the column. 

25 Similar readout sequences can be devised in accordance with the present invention for 

color filter patterns other than Bayer and for color systems other than the primary one. 

Figure 6 illustrates one embodiment of the on-the-fly color interpolation process 
in a fill! resolution mode. In a process block 202, the illustrated system 140 may 
simultaneously read out a group of 2 by 2 pixels that contains the Bayer pattem core. 

30 In a process block 203, the system 140 may perform several acts substantially 

simultaneously (in parallel). In blocks 204, 210, the system 140 reads the 
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corresponding red and the blue components 190, 196 that reside in two consecutive 
columns 148, 148" (Figure 4) and amplifies accordingly via the programmable red and 
blue gain amplifiers 156, 172. In blocks 206-208, the system 140 reads the two green 
components 192, 194 that reside in two consecutive columns 148, 148" and sums the 
5 values with the sunoming ampHfier 154. The gain from the summing amplifier 154 is 
adjusted accordingly via the programmable gain amplifier 166. The programmable 
readout control 146, 180 ensures that the appropriate switches are closed to allow the 
correct readout sequence as described in Figure 5. 

In Figure 6, the acts in block 203 (amplifying and outputting a red value, 

10 ampUfying and outputting a green value and amplifying and outputting a blue value) 

are performed substantially simultaneously. In another embodiment, the acts of block 
203 are performed in a sequence, e.g. the system 140 amplifies and outputs a red value, 
then amplifies and outputs a green value, and then amplifies and outputs a blue value. 

In Figure 6, the system 140 advances to the next column in a block 212, nev/ 

15 COLUMN = old COLUMN +1. In a block 214, the system 140 determines whether 

the column readout control 146 is exceeding the last Red pixel in a row. If yes, then the 
process proceeds to a block 216, and the row readout control 180 advances to the next 
row, new ROW = old ROW + 1. If not, then the process loops back to block 202. In a 
block 218, the system determines whether the row readout control 180 is exceeding the 

20 last Red pixel in a frame. If yes, then the process stops in a stop block 220 and waits 

for further commands. If not, then the process proceeds to block 202. 

Figure 7 illustrates one embodiment of a readout sequence for on-the-fly color 
interpolation in a sub-sampling mode. In Figure 7, the sequence accommodates an 
RGB Bayer pattem color filter 230 in a non-window mode with sub-sampling. In 

25 Figure 7, the sub-sampling ratio is 1:4 horizontally and 1:4 vertically. Each RGE$ 

triplet that is needed for a color pixel representation is produced from a group 232 of 
4x4 pixels that contains 4 Bayer pattem cores. The 8 green pixels are averaged to 
produce the green component, the 4 red pixels are averaged to produce the red 
component, and the 4 blue pixels are averaged to produce the blue component. 

30 Specifically, as illustrated in Figure 7, the red component comprises 

[(R(0,0) + R(2,0) + R(0,2) + R(2,2))/4], 
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the green component comprises 

[(G(1,0) + G(3,0) + G(OJ) + 0(2,1) + 0(1,2) + 0(3,2) + 0(0,3) + 0(23))/8], 
and the blue component comprises 

[(B(l,l) + B(3,l) + B(13) + B(3,3))/4] 
5 Similar readout sequences can be devised for color filter patterns other than Bayer and 

for color systems other than the primary one. 

Figure 8 illustrates one embodiment of a CMOS integrated circuit color imaging 
system 240 that supports on-the-fly color interpolation in a programmable pixel readout 
mode, such as a sub-sampling mode. The structure of the system 240 is substantially 

10 similar to the system 140 of Figure 4, except that the system 240 also includes a red 

summing ampUfier 242 and a blue summing amplifier 244. The function of these 
additional amplifiers is described below. Figure 8 presents an implementation for an 
ROB Bayer pattern on-the-fly color interpolation where windowing is not active and 
sub-sampling is active. However, the system 240 is not limited to a Bayer pattern 

15 and/or the primary color scheme. Nor is the system 240 limited in its mode of 

operation. The system 240 can be realized to support both window and sub-sampling 
via the programmable readout control circuitry 146, 180, the summing amplifier(s) 242, 
154, 244, and the programmable gain ampUfiers 156, 166, 172. 

Figure 9 illustrates one embodiment of the on-the-fly color interpolation process 

20 in a sub-sampling mode, as described with reference to Figures 7 and 8. Figure 9 

illustrates sub-sampling a ROB Bayer color filter 230 (Figure 7) using a ratio of 4:1 
horizontally and 4:1 vertically. In a block 250, a group 232 (Figure 7) of 4 by 4 pixels 
that contains Bayer pattem cores is read out simultaneously. 

In a process block 253, the system 240 may perform several acts substantially 

25 simultaneously (in parallel). In blocks 252, 254, the four red pixels shown in sub- 

sample 232 of Figure 7 are summed and amplified by the red amplifiers 242, 156 of 
Figure 8. In blocks 256, 258, the 8 green pixels that reside in four consecutive columns 
(Figure 7) are summed via the green summing amplifier 154 and amphfied by the green 
amphfier 166 (Figure 8). In blocks 260, 262, the four blue pixels shown in sub-sample 

30 232 of Figure 7 are summed and amphfied by the blue amphfiers 244, 172 of Figure 8. 

The programmable readout control 146, 180 of Figure 8 ensures that the appropriate 
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switches are closed to allow the correct readout sequence as described in Figures 7 and 
9. 

In Figure 9, the acts in block 253 are performed substantially simultaneouslj^ 
In another embodiment, the acts of block 253 are performed in a sequence, e.g. the 
5 system 240 sums, amplifies and outputs a red value, then sums, amplifies and outputs a 

green value, and then sums, amplifies and outputs a blue value. 

In Figure 9, the system 240 advances to the next 4x4 block horizontally in a 
block 264, new COLUMN = old COLUMN + 4. In a block 266, the system 240 
determines whether the column readout control 146 is exceeding the last 4x4 block in a 

10 row. If yes, then the process proceeds to a block 268, and the system 240 advances 
vertically to the next 4x4 block, new ROW - old ROW + 4. If not, then the process 
loops back to block 250. In a block 270, the system 240 determines whether the row 
readout control 180 is exceeding the last 4x4 block in a fi^ame. If yes, then the process 
stops in a stop block 272 and waits for fiorther commands. If not, then the process 

1 5 proceeds to block 250. 

Figure 10 illustrates a one embodiment of window readout mode. For 
simplicity, sub-sampling and/or on-the-fly color interpolation are not included in the 
embodiment shown in Figure 10. In window mode, a sub-region 280 of the pixel array 
exposed area 130 is readout. In the embodiment shown in Figure 10, the pixel array 

20 exposed area size is n by m, the window size is chosen to be n - 4 by m - 6, and the 

window origin is chosen to be (4,4). Thus, the sub-region 280 comprises pixels in a 
rectangle defined by four pixels R(4,4), G(m-3,4), G(4,n) and B(m-3,n) at the four 
comers. 

A window mode is essentially a cropping operation that produces a smaller area 
25 of interest. The readout is faster and provides a higher fi-ame rate than the system 240 

of Figure 8. The system 240 of Figure 8 may be modified to support window readout 
mode. Window mode may be used in digital cameras for exposure and focus 
calculations, for electronic zoom and more. 

Figure 11 illustrates one embodiment of a column readout control 146 (Figure 
30 4) that supports on-the-fly color interpolation in a Ml resolution mode as illustrated in 

Figure 5. The readout control of the system 146 advantageously allows the 
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simultaneous readout of a 2x2 pixel block. The readout control comprises a column 
readout control 146 that provides simultaneous readout of two columns and a row 
readout control 180 that provides a simultaneous readout of two rows. 

The column readout control 146 includes a shift register that is responsible for 
5 closing the switches of the column buffers two at a time. The pixel clock 144 clocks 
the shift register 146. The line sync signal 142 is feeding the shift register 146 and is 
multiplied by two to allow signal width of two consecutive bits. Thus, the shift register 
146 closes two consecutive column buffer switches coupled to lines 148, 148' 
simultaneously. 

10 The row readout control 180 (Figure 4) is substantially similar except the shift 

register clock of the row readout control 180 utiUzes a Une sync 184 instead of the pixel 
clock, and the shift register input data utilizes a fi-ame sync 182 instead of the line sync 
signal. 

Figure 12 illustrates one embodiment of a novel readout control that supports 
15 on-the-fly color interpolation in sub-sampling mode shown in Figure 8, as well as the 

full resolution mode shown in Figure 5. Figure 12 demonstrates the column readout 
control 146. The sub-sampling modes chosen for the example are: 1:2, 1:4, 1:8 and 
1:16. The readout control of Figure 12 also supports overlap such as the case in the fiiU 
resolution mode (Figure 5). The shift register that is responsible for closing the column 
20 buffer switches is divided into 16-bit registers 318, 318', 318" that are clocked by the 

pixel clock 324. 

In Figure 12, each register 318 is loaded with a bit pattern that is produced by 
the pattern generator 290. In a fiill resolution mode, the pattern generator 290 shifts a 
pattern 292 (provided by the initial pattern signal line) left by 1 during each cycle and 

25 causes an overlap. For a sub-sampling ratio of 1 :4, "1 1 1 1" is the pattem 292 provided 

on an initial pattem line and loaded since four columns are read simultaneously. In 
each cycle, the pattem is shifted left and selected by the multiplexors 304, 308. For 
example, a shift amount of 4 is used for sub-sampling 1 :4 without overlap. The down 
coimter 314 is responsible for counting pixel clocks for each 16-bit register 318, e.g. for 

30 sub-sampling of 1:4 without overlap, the counter 314 counts 4 times from the line sync 

322. Then the counter 314 is reloaded by the register 312, and the next 16-bit register 
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318' is enabled by the latch 316 and the initial pattern is reloaded by the 16-bit register 
308. 

The row readout control 180 is similar to the column readout control 146 shown 
in Figure 12 except the clock 324 utilizes a line sync 184 instead of the pixel clock, and 
5 the initializing signal 322 utilizes a frame sync 182 instead of the line sync signal. 
Color Compensation 

As previously discussed, another aspect of the invention relates to color 
compensation (correction or maximization). The color compensation process 
compensates for differences in the response of various color filters and variations 

10 within the integrated circuit sensor array, such as process, materials, temperature or 
manufacturing. The color compensation process of the present invention improves 
color dynamic range, decreases computations and hardware and improves speed of 
operation. The color compensation process of the present invention is advantageously 
performed in the analog domain before an analog-to-digital conversion. 

15 Figure 13 illustrates one embodiment of a CMOS integrated circuit color 

imaging system 330 of the present invention with analog color correction. The system 
330 provides a programmable pixel readout mode that allows four independent readouts 
and four programmable gain ampUfiers 156, 172, 342, 352 for amplifying the four 
output paths (red, blue, even green, odd green) separately. Figure 13 presents an 

20 implementation for an RGB Bayer pattern. However, the system 330 is not limited to a 

Bayer pattern and/or the primary color scheme. In Figure 13, the system 330 employs 
readout control circuitry 146, 334 with separate paths for even and odd rows and 
columns 174, 178, 348, 350, and programmable gain ampUfiers 156, 172, 342, 352. 
Thus, the system 330 provides four separate, simultaneous output paths with potentially 

25 different amplified gains. 

The system 330 is not restricted to a particular type of transfer function with 
respect to the programmable gain amphfiers 156, 172, 342, 352. In one embodiment, 
each of the four programmable gain ampUfiers 156, 172, 342, 352 implement different 
transfer functions, such as a log or an exponent function where the power value is 

30 programmable, as shown in Figure 17. Thus, the different transfer functions 

advantageously allow different levels of color compensation for different colors. In one 
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embodiment, the transfer functions may be modified in real-time according to the level 
of compensation needed to accommodate temperature, differences in the response of 
various color filters and variations within the integrated circuit sensor array, such as 
process, materials or manufacturing. In another embodiment, the transfer fimctions maty 
5 be pre-set at the manufacturer to accommodate variations within the integrated circuit 

sensor array, such as process, materials or manufacturing. 

Figure 14 provides one embodiment for the analog color correction process 
performed by the system 330 of Figure 13. In a block 360, the system 330 (Figure 13) 
reads a pixel value. In a block 363, the system 330 may perform several acts 

10 simultaneously (in parallel). Blocks 362, 368 and 372 determine the color of the pixel 

value. In a block 364, the red pixel 190 that resides in an even column and even row is 
amphfied via the red programmable amplifier 156. In a block 366, the green pixel 192 
that resides in an even column and odd row is amphfied via the even green 
programmable amplifier 342. In a block 370, the green pixel 194 that resides in an odd 

15 column and even row 194 is amphfied via the odd green amphfier 352. In a block 374, 

the blue pixel 196 that resides in an odd column and odd row is amphfied via the blue 
programmable amphfier 172. The acts in block 363 are performed substantially 
simuhaneously. In another embodiment, the acts of block 363 are performed in a 
sequence. 

20 In a block 376, the system 330 advances to the next column, new COLUMN 

old COLUMN +1. In a block 378, the system 330 determines whether the system 
readout is exceeding the last pixel in a row. If not, the system 330 returns to process 
block 360. If yes, the system 330 advances to the next row, new ROW = old ROW + 1 , 
in a block 380, In a block 382, the system 330 determines whether the system readout 

25 is exceeding the last row in a frame. If not, the system 330 returns to process block 360. 

If yes, the system 330 stops in a stop block 384 and waits for fiirther commands. 

In addition to the significant innovations described above, one embodiment of 
the color compensation process of the present invention also improves white balancing, 
which is used to compensate colors for different illumination temperatures. 

30 Figure 15 illustrates one embodiment of a CMOS integrated circuit color 

imaging system 390 that accommodates analog white balance with programmable pixel 
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readout modes, multiplexor(s) 418, programmable gain amplifiers 398-414, and 
summing amplifiers 392, 394 and 396. Figure 15 presents an implementation for sin 
RGB Bayer pattern. However, the system 390 is not limited to a Bayer pattern and/or 
the primary color scheme. In Figure 15, the system 390 employs readout control 
5 circuitry 146, 180 with separate paths for even and odd rows 186 and columns 148, a 

multiplexor 418 for selecting the appropriate green path, programmable gain amplifiers 
398-414, and summing amplifiers 392, 394 and 396. The multiple amplifier gain stages 
provide convolutions associated with white balance. 

Figures 16, 16A and 16B (hereinafter referred as "Figure 16") provide one 

10 embodiment of an analog white balance process performed by the system 390 of Figui'e 
15. In a block 420, the system 390 reads a 2x2 pixel block. In a process block 423, the 
system 390 may perform several acts substantially simultaneously (in parallel). In 
blocks 422 and 430, the system 390 determines the colors of the pixels. In blocks 424- 
428, the red pixel 190 is amphfied by 3 red programmable amphfiers 398-402 (Figui-e 

15 15). In blocks 440, 447 and 444, the blue pixel 196 is amphfied by 3 blue 

programmable amphfiers 410-414. In a block 432, the green pixel 192 that resides in an 
even column and odd row is multiplexed by the multiplexor 418 with the green pixel 
194 that resides in an odd column and even row to produce one green value that is 
amplified by 3 green amplifiers 404-408. In blocks 446, 448 and 450, each of the 

20 amplified red components is summed with the corresponding amplified green and 

amplified blue components via 3 summing amplifiers 392, 394 and 396 (Figure 15), and 
thus producing new red, green and blue components that are results of RGB 
convolutions. 

In a block 452, the system 390 advances to the next column, new COLUMN = 
25 old COLUMN +1. In a block 454, the system 390 determines whether the system 
readout is exceeding the last pixel in a row. If not, the system 390 returns to process 
block 420. If yes, the system 390 advances to the next row, new ROW = old ROW + 1, 
in a block 456. In a block 458, the system 390 determines whether the system readout 
is exceeding the last row in a fi-ame. If not, the system 390 returns to process block 420. 
30 If yes, the system 390 stops in a stop block 460 and waits for fiirther commands. 
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The acts in block 423 are performed substantially simultaneously. For example, 
the decision blocks 422 and 430 may be performed in parallel by reading two or more 
pixels simultaneously. As another example, blocks 424, 426, 428, may be performed in 
parallel by themselves or with blocks 440, 447, 444 and/or blocks 434, 436, 438. In 
5 other embodiments, the blocks may be performed in a sequence. The system 390 may 
further comprise storage circuits to store the values of the amplified red, green and blue 
components before they are summed in blocks 446, 448 and 450. 

Figure 17 presents embodiments of optional transfer functions 470, 472, 474, 
such as an exponent transfer function, for the programmable gain amplifiers 398-414 of 

10 Figure 15. Each amplifier can implement a different transfer function and thus optimize 
each color for maximum dynamic range. 

Figures 18, 18A and 18B (hereinafter referred as 'Tigure 18") provide another 
embodiment of the process shown in Figure 16. The process of Figure 18 provides 
white balance and on-the-fly color interpolation. The process may be performed by the 

15 system 390 of Figure 19, The process of Figure 18 is similar to the process shown in 

Figure 16, except the two green pixels are averaged by a summing amphfier 482 
(Figure 19) in a block 432 of Figure 18. In Figure 18, like the process in Figure 16, 
some of the process blocks are performed in parallel. In other embodiments, the blocks 
may be performed in a sequence. 

20 Fixed Pattern Noise Reduction 

Another aspect of the invention relates to a fixed pattern noise reduction 
apparatus and process. The fixed pattern noise reduction process reduces noise related 
to pixel-to-pixel variation. This variation is primarily due to dark current leakage, 
which may be integrated together with the signal and hence contaminate the signal. The 

25 dark current leakage may be due to thermal generation in the neutral bulk material, in 

the depletion region and due to surface states. The dark current level may vary between 
pixels and may be particularly noticeable between columns due to column buffers. 

The fixed pattern noise reduction process of the present invention allows 
increased dynamic range (high image quality), high speed of operation and reduced 

30 computational complexity. The fixed pattern noise reduction of the present invention is 

advantageously performed in the analog domain before an analog-to-digital conversion. 
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Figure 20 illustrates one embodiment of an imaging system that supports fixed 
pattern noise reduction. Specifically, Figure 20 illustrates one embodiment of a CMOS 
integrated circuit color imaging system 500 that supports on-the-fly fixed pattern noise 
reduction. In the illustrated embodiment, additional circuitry is not needed for post- 
5 processing in order to provide the fixed pattern noise reduction. The system 500 

comprises a column readout shift register 146, a row readout shift register 180, a dark 
row readout control or path 492, a row of dark pixel sensor elements (dark row) 490, a 
first row of exposed pixel sensor elements 186, a second row of exposed pixel sensor 
elements 186', a summing amplifier 496, a programmable gain amplifier 498 and a 

1 0 plurality of switches 508. 

In one embodiment, the dark pixels are deposited with an opaque mask layer 
and thus are not exposed to light. A row of pixels with dark pixels may be referred to 
as a "dark row." The system 500 presents an implementation for a red/green/blue 
(RGB)(primary color scheme) in a Bayer pattern with on-the-fly fixed noise reduction 

1 5 when windowing and sub-sampling are not active. 

The system 500, however, is not Umited to a Bayer pattern and/or the primary 
color scheme. Other embodiments may use other color schemes and/or other color 
patterns. Nor is the system 500 limited in its mode of operation. For example, the 
system 100 of Figure 1 may be modified to support windowing and/or sub-sampling. 

20 In Figure 20, two pixels in the same column are read out simultaneously in 

response to signals firom the column shift register 146, the row shift register 180 and the 
dark row readout control 492. One pixel is read fi^om the dark row 490 and another 
pixel is read fi-om the first exposed row 186. The dark pixel value is then subtracted 
from the exposed pixel value by the summing amplifier 496, thereby providing fixed 

25 pattern noise reduction. Additional gain is applied to the new value by the 

programmable gain amphfier 498. The dark row 490 may be re-used for each exposed 
row 186, 186'. The programmable readout control components 146, 180 ensure that the 
appropriate switches 508 are closed to allow the correct readout sequence. 

Figure 21 illustrates one embodiment of a fixed pattern noise reduction process 

30 that may be performed by the system 500 of Figure 20. Specifically, Figure 21 

illustrates one embodiment of an on-the-fly fixed pattern noise reduction process 
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utilizing a single dark row 190. In Figure 21, in a process block 520, the system 500 
reads two pixels in a current column: a dark pixel in the dark row 490 and an exposed 
pixel in an exposed row 186. In a block 522, the summing amplifier 496 subtracts a 
value associated with the dark pixel from a value associated with the exposed pixel. In 
5 a block 524, the system 500 clips the difference found in block 522 to zero if the 

difference is negative. In a block 526, the system 500 advances to the next colunm of 
the system 500 in Figure 20, new COLUMN = old COLUMN + 1. 

In a decision block 528, the system 500 determines whether the column shift 
register 146 has reached the last pixel in a row 186. If the column shift register 146 has 

10 not reached the last pixel in a row 186, then the system 500 returns to process block 520 
to process the next colunm. If the column shift register 146 has reached the last pixel in 
a row 186, then the system 500 advances to the next row 186' in a process block 530, 
new ROW = old ROW +1. In a decision block 532, the system 500 determines 
whether the row shift register 180 has reached the last row in a frame of pixel sensor 

15 elements. If the row shift register 180 has not reached the last row in a frame of sensor 

elements, then the system 500 returns to process block 520 to process the first column 
in the next row 186*. If the row shift register 180 has reached the last row in the frame, 
then the system 500 may wait for a further command in an end block 534. 

Figure 22 illustrates another embodiment of an imaging system 540 that 

20 supports fixed pattern noise reduction. Specifically, Figure 22 illustrates one 

embodiment of a CMOS integrated circuit color imaging system 540 that supports on- 
the-fly fixed pattern noise reduction with two dark rows. The imaging system 540 uses 
more than one dark rows for improved quahty. Each image pixel value is preferably 
produced from a combination of a current exposed pixel and a dark current value that is 

25 the average of two dark pixels residing in the same column. 

In Figure 22, the system 540 comprises a column readout shift register 146, a 
row readout shift register 180, a dark row readout control or path 492, a first row of 
dark pixel sensor elements (first dark row) 490, a second row of dark pixel sensor 
elements (second dark row) 544, a first row of exposed pixel sensor elements 186, a 

30 second row of exposed pixel sensor elements 186', a first summing amplifier 542, a 

second sununing amphfier 496, a programmable gain amphfier 498 and a plurahty of 
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switches 508. The system 540 presents an implementation for a red/green^lue 
(RGB)(primary color scheme) in a Bayer pattern with on-the-fly fixed noise reduction 
when windowing and sub-sampling are not active. 

In another embodiment, the dark rows 490, 544 and/or individual dark pixels are 
distributed over the imaging array. 

The system 540 is not limited to a Bayer pattern and/or the primary color 
scheme. Other embodiments may use other color schemes and/or other color patterns. 
Nor is the system 540 limited in its mode of operation. For example, the system 540 of 
Figure 22 may be modified to support window and sub-sampling. 

In Figure 22, three pixels in the same column are read out simultaneously by the 
column shift register 146, the row shift register 180 and the dark rows readout path 492. 
Two pixels are read from the two dark rows 490, 544 and a third pixel is read from the 
first exposed row 186. The two dark pixels are then averaged by the first summing 
amplifier 542 to produce an average dark value. The average dark value is then 
subtracted from the exposed pixel by the second summing amplifier 496, Additional 
gain is appUed to the new value by the programmable gain amplifier 498. The dark row 
490 is re-used for each exposed row 186. The programmable readout control 
components 146, 180 ensure that the appropriate switches 508 are closed to allow the 
correct readout sequence. 

In Figure 22, the system 540 has two dark rows 490, 544 for producing average 
dark values. In other embodiments, there are more than two dark rows which are 
averaged. The fixed pattern noise reduction system of the present invention is not 
limited to two dark rows nor to any other particular number. The system 540 in Figure 
22 can be modified to average any number of dark rows prior to the subtraction from 
the exposed pixel by the summing amplifier 496. 

Figure 23 illustrates one embodiment of an on-the-fly fixed pattern noise 
reduction process utilizing multiple dark rows 490, 544. The process of Figure 23 may 
be performed by the system 540 of Figure 22. In Figure 23, in a process block 550, the 
system 540 reads three pixels in a current column: a dark pixel from each of the two 
dark rows 490, 544 and an exposed pixel in an exposed row 186. In a process block 
552, the first summing amphfier 552 averages the values associated with the two dark 
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pixels. In a process block 554, the second summing amplifier 496 subtracts the 
averaged dark pixel value from a value associated with the exposed pixel. In a process 
block 546, the system 540 clips the difference found in block 554 to zero if the 
difference is negative. In a process block 558, the system 540 advances to the next 
column. 

In a decision block 560, the system 540 determines whether the column shift 
register 146 has reached the last pixel in a row 186. If the column shift register 146 has 
not reached the last pixel in a row 186, then the system 540 returns to process block 550 
to process the next column. If the colunrn shift register 146 has reached the last pixel in 
a row 186, then the system 540 advances to the next row 186' in a process block 562. In 
a decision block 564, the system 540 determines whether the row shift register 180 has 
reached the last row in a frame of pixel sensor elements. If the row shift register 180 
has not reached the last row in a frame of sensor elements, then the system 540 returns 
to process block 550 to process the first column in the next row 105'. If the row shift 
register 180 has reached the last row in the frame, then the system 540 may wait for a 
further command in an end block 566. 

Bad Pixel Correction 

In one embodiment, the imaging system contains a signal processing circuit 624 
that allows bad pixel correction as shown in Figure 24. Figure 24 demonstrates an 
example of a CMOS integrated circuit imaging device implementation that 
accommodates on-the-fly bad pixel coixection. The illustration presents an 
implementation for a monochrome on-the-fly bad pixel correction when windowing and 
sub-sampling are not active. However, the present invention is not limited to a 
monochrome imager, nor is hmited in its mode of operation and can be reaUzed to 
support window and sub-sampling. 

In the embodiment shown in Figure 24, a monochrome imaging system 600 
employs a CMOS integrated circuit. The bad pixel correction system may be applied to 
CCD integrated circuits and/or different color fifters. 

Figure 24 illustrates one embodiment of a CMOS integrated circuit imaging 
system 600 that accommodates on-the-fly bad pixel correction. In Figure 24, the 
system 600 comprises a row shift register 180, a column shift register 146, buffers 188, 
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188', 152, row control lines 186, 186', column control lines 602, 602', 602", a delay 
604, a mdtiplexer 606, an amplifier 608, a video out line 610, an external memory 
control 612, a row comparator 616, a column comparator 614, an AND gate 622, a row 
comparator 618, a column counter 620, a row control line 630 and a column control line 
632. Other circuit configurations may be implemented in accordance with the present 
invention. In another embodiment of Figure 24, a pipeline delay of two or more pixels 
is utihzed. In a pipeline approach described below, a delay is not needed because one 
or more lines are stored such that the previous pixels are readily available for bad pixel 
correction. 

In Figure 24, the row control line 630 inputs the total number of rows into the 
row counter 618, and the column control line 620 inputs the total number of columns 
into the column counter 620. The color imaging system 600 provides an on-the-fly bad 
pixel correction process that is performed during the analog sensor readout stage. The 
signal processing circuit 624 identifies the addresses of the bad pixels via a list stored in 
an external memory accessed by the external memory control 612 and corrects any bad 
pixels on-the-fly. The list may be stored in a memory either on-chip or off-chip. 

Figure 25 illustrates a flowchart for the on-the-fly bad pixel correction of Figure 
24. The detection of bad pixels is performed off-line and a bad pixel hst is stored in an 
external memory which is accessed by the external memory control 612. The 
correction process utilizes the Ust and replaces a defect pixel with its adjacent neighbor. 

In a block 630 of Figure 25, the circuit 600 reads consecutive horizontal pixels 
and amphfies the pixel readouts with an amphfier 608. The delay 604 provides a 
temporary pixel value storage. In a block 632, the row address comparator 616 and the 
column address comparator 614 compare the readout pixel address fi-om the row and 
column counters 618, 620 with the stored Hst of defective pixel addresses. In a decision 
block 634, the circuit 624 determines if the current pixel is a bad one. If the 
comparison is positive, the output of the AND gate is 1, and the current pixel is 
replaced by the previous adjacent pixel that is stored in the delay 604, as shown in a 
block 636. If the comparison is negative, the output of the AND gate is 0, and the 
column shift register 146 advances to the next column in a block 638. The blocks 640- 
646 are substantially similar to the blocks 214-220 in Figure 6 described above. 
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The on-the-fly bad pixel correction embodiment may be combined with various 
readout modes such as windowing and sub-sampling, and additional signal processing 
such as on-the-fly color interpolation. The bad pixel correction system of the present 
invention is not limited to monochrome imaging systems and is applicable to integrated 
circuit color imaging systems as well. The color imaging system may utiUze any color 
filter scheme such as the primary color system (RGB) in a Bayer pattern, a 
complementary color system or others. The imaging system may also use a non- 
rectangle matrix of pixels. A similar implementation can provide on-the-fly bad pixel 
correction for color imaging systems and/or different pixel topography via modification 
to the readout control, the pixel neighborhood size and configuration. For example, a 
color imaging system with an RGB Bayer pattern may use 3 horizontally consecutive 
pixels, i.e., pipeline delay of 3 pixels and 2 registers for temporary storage. So each 
color component when defective, can be replaced by the corresponding pixel with the 
same color component. 

In other embodiments of the systems illustrated in Figures 4, 8, 13,15, 19, 20, 
22, the systems may further include one or more rows and/or columns of capacitors to 
read and store pixel sensor values to accommodate serial readout of pixels. Using 
capacitors to store values reduces the number and complexity of the lines (buses) 
coupled the pixel sensor elements. The size of the pixel sensor elements, the size of the 
sensor array, the space on the chip and the placement of the components are design 
choices of the manufacturer. 

The present invention does not require the integrated color imaging system to 
employ a simultaneous readout of the n-by-n pixel block. A ^pipeline' approach may be 
utilized instead of a parallel readout. The pipeline approach uses one or more analog 
line storage units, e.g., capacitors. For example, in one embodiment using the Bayer 
color pattern, two line storage units are used. The first of two consecutive lines that is 
readout fi-om the array is stored in the first line storage unit. The second line is 
averaged with the stored Une to produce the RGB triplets, while a "first" line of the next 
two consecutive lines is readout and stored in the second line storage unit and so on. 
Thus, the two line storage units are used in a 'ping pong' fashion. 
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The invention may be embodied in other specific forms without departing fi-om 
its spirit or essential characteristics. The described embodiment is to be considered in 
all respects only as illustrative and not restrictive and the scope of the invention is, 
therefore, indicated by the appended claims rather than the foregoing description. All 
changes which come within the meaning and range of equivalency of the claims are to 
be embraced within their scope. 
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WHAT IS CLAIMED: 



5 LA flexible pixel sensor element control system configured to readout and 

process analog values from a plurality of pixel sensor elements, the system comprising: 
an array of pixel sensor elements; and 

a readout and processing circuit configured to readout and process a 
plurality of analog values associated with a plurality of pixel sensor elements 
10 within the array, the readout and processing circuit being configured to readout 

and process the analog values in a first mode and in a second mode. 

2. The system of Claim 1, wherein the readout and processing circuit is 
adapted to read a pluraUty of pixel sensor elements in parallel. 

3. The system of Claim 1, further comprising: 

15 a first analog line storage unit, the first analog line storage unit being 

adapted to store a first line readout from the array; and 

a second analog line storage unit, the second analog line storage unit 
being adapted to store a third line readout from the array, wherein the readout 
and processing circuit averages a second consecutive line readout from the array 

20 with the first hne readout stored in the first analog hne storage unit to produce a 

first red-green-blue (RGB) triplet, the readout circuit and processing averaging a 
fourth consecutive line readout from the array with the third line readout stored 
in the second analog line storage xmit to produce a second RGB triplet. 

4. The system of Claim 1, wherein the pixel sensor elements form a portion 
25 of a charge coupled device. 

5. The system of Claim L wherein the pixel sensor elements form a portion 
of a complementary metal oxide semiconductor device. 

6. The system of Claim 1, wherein the pixel sensor elements are organized 
in a rectangular matrix. 

30 7. The system of Claim 1, wherein the first mode comprises a full 

resolution readout mode. 
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8. The system of Claim 1, wherein the first mode comprises a sub-sarapHng 
readout mode. 

9. The system of Claim 1, wherein the first mode comprises a window 
readout mode. 

10. The system of Claim 1, further comprising a color filter overlaying at 
least a portion of the pixel sensor elements. 

11. The system of Claim 10, wherein the color filter includes the colors of 
red, blue and green in a predefined pattern. 

12. The system of Claim 10, wherein the color filter includes the colors of 
yellow, cyan and magenta in a predefined pattern. 

13. The system of Claim 10, wherein the color filter comprises a Bayer color 

pattern. 

14. The system of Claim 1, fiirther comprising a micro-lenses layer. 

15. The system of Claim 1, fiirther comprising amplifiers adapted to amphfy 
the analog values readout and processed by the readout and processing circuit. 

16. The system of Claim 15, wherein the programmable gain amplifiers are 
implemented as a separate stage. 

17. The system of Claim 15, wherein the programmable gain amplifiers are 
contained within a pixel circuitry of the array. 

18. The system of Claim 15, wherein the programmable gain amplifiers are 
within a plurality of column buffers. 

19. The system of Claim 1, wherein a first gain amplifier ampHfies a first 
analog color component a first amount and a second amplifier amphfies a second analog 
color component a second amount, the first and second analog color components being 
readout by the readout and processing circuit. 

20. The system of Claim 19, wherein the amplifiers are programmable gain 
amplifiers adapted to be adjusted by a controller. 

21. The system of Claim 19, wherein the first gain amplifier provides a first 
transfer fimction for the first color component and the second gain ampUfier provides a 
second transfer fimction for the second color component. 
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22. The system of Claim 15, wherein at least one of the amplifiers is a 
summing amplifier that sums the analog values of two or more pixel sensor elements. 

23. The system of Claim 1, further comprising a television coupled to said 
readout and processing circuit. 

24. The system of Claim 1, further comprising a personal computer coupled 
to said readout and processing circuit. 

25. The system of Claim 1, further comprising a display coupled to said 
readout and processing circuit. 

26. The system of Claim 1, further comprising a camera coupled to said 
readout and processing circuit. 

27. A flexible pixel sensor element control system that processes of a 
plurality of pixel sensor elements, the system comprising: 

an array of pixel sensor elements; and 

a control circuit, wherein the control circuit reads out and averages a first 
analog value readout firom a pixel sensor element of a first color with a second 
analog value readout fi-om a pixel sensor element of a second color to produce 
an average readout value. 

28. The system of Claim 27, wherein the control circuit is adapted to read a 
plurality of pixel sensor elements in parallel. 

29. The system of Claim 27, further comprising: 

a first analog line storage unit, the first analog line storage unit being 
adapted to store a first line readout fi-om the array; and 

a second analog line storage unit, the second analog line storage unit 
being adapted to store a third line readout firom the array, wherein the readout 
and processing circuit averages a second consecutive line readout fi-om the array 
with the first line readout stored in the first analog line storage unit to produce a 
first red-green-blue (RGB) triplet, the readout circuit and processing averaging a 
fourth consecutive line readout from the array with the third line readout stored 
in the second analog hne storage unit to produce a second RGB triplet. 

30. The system of Claim 27, wherein the control circuit reads out and 
averages the first and second analog values on-the-fly. 
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31. The system of Claim 27, further comprising gain ampHfiers ampUfying 
the average readout value. 

32. A method of processing in parallel a plurality of pixel sensor elements, 
the method comprising: 

5 exposing an array of pixel sensor elements to hght; 

selecting a plurality of pixel sensor elements from an array of pixel 
sensor elements; and 

reading in parallel a plurality of analog values associated with the 
plurality of pixel sensor elements. 
10 33. A method of averaging two or more pixel sensor elements, the method 

comprising: 

exposing an array of pixel sensor elements to hght; 
selecting a plurality of pixel sensor elements from an array of pixel 
sensor elements; and 

15 averaging a first analog value associated with a first pixel sensor element 

and a second analog value associated with a second pixel sensor element within 
the array to produce an average readout value. 

34. A pixel sensor element control system configured to readout and process 
analog values from a plurahty of pixel sensor elements, the system comprising: 
20 an array of pixel sensor elements; 

a readout and processing circuit configured to readout and process a 
plurahty of analog values associated with a plurality of pixel sensor elements 
within the array, the readout and processing circuit being configured to readout 
and process the analog values in a first mode and in a second mode. 
25 a comparator circuit adapted to compare an address of a pixel sensor 

element currently being read by the readout controller with a stored hst of 
defective pixel sensor addresses; and 

at least one delay element for storing at least one previous analog pixel 
value read by the readout and processing circuit, wherein if the address of the 
30 current pixel sensor element matches a defective pixel address in the stored hst, 

the readout and processing circuit reads the previous analog pixel value. 
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35. A method of correcting defective pixel sensor elements during an analog 
readout, the method comprising: 

comparing an address of a pixel sensor element currently being read by a 
readout control circuit with a stored list of defective pixel sensor addresses; 

storing at least one previous analog pixel value read by the readout 
control circuit, wherein if the address of the current pixel sensor element 
matches a defective pixel address in the stored list, the readout control circuit 
reads the previous analog pixel value. 
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METHOD AND APPARATUS FOR CONTROLLING PIXEL SENSOR 

ELEMENTS 



10 



Abstract of the Disclosure 

An imaging method and system that flexibly accesses light sensor elements and 
processes imaging signals. The imaging system comprises an array of pixel sensor 
cells, an array controller and a readout control circuit. 
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READ 2x2 PIXEL BLOCK: 
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READ 4x4 PIXEL BLOCK: 
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READ ONE PIXEL VALUE 



J62 




USE THE RED OUTPUT PATH 
AMPLIFY VALUE VIA RED GAIN AMPLIFIER 
OUTPUT RED VALUE 
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USE THE GREEN EVEN OUTPUT PATH 
AMPLIFY VALUE VIA GREEN EVEN GAIN AMPLIFIER 
OUTPUT GREEN EVEN VALUE 



USE THE GREEN ODD OUTPUT PATH 
AMPLIFY VALUE VIA GREEN ODD GAIN AMPLIFIER 
OUTPUT GREEN EVEN VALUE 



USE THE BLUE OUTPUT PATH 
AMPLIFY VALUE VIA BLUE GAIN AMPLIFIER 
OUTPUT BLUE VALE 



'J 74 




F/G. 



F/G, 75A 



r/a WB 



F/O. 76 



READ 2x2 PiXEL BLOCK: 
RG or GR or GB or BG 
GB BG RG GR 
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AMPLIFY VALUE VIA 
FIRST RED GAIN AMPLIFIER 

AMPLIFY VALUE VIA 
SECOND RED GAIN AMPLIFIER. 

AMPLIFY VALUE VIA 
THIRD RED GAIN AMPLIFIER 



SELECT THE GREEN PIXEL THAT 

RESIDES IN CURRENT ROW 
OMIT THE GREEN PIXEL THAT 
RESIDES IN THE FOLLOWING ROW 
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AMPLIFY VALUE VIA 
FIRST BLUE GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
SECOND BLUE GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
THIRD BLUE GAIN AMPLIFIER 
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ADVANCE TO NEXT COLUMN: 
C0LUMN=C0LUMN+1 




ADVANCE TO NEXT ROW: 
R0W=R0W+1 
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exceeding last row 
^with red in a frame ^ 
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AMPLIFY VALUE VIA 
FIRST GREEN GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
ECOND GREEN GAIN AMPLIFIE 



AMPLIFY VALUE ViA 
THIRD GREEN GAIN AMPLIFIER 
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AMPLIFIED RED+ 
AMPLIFIED GREENE- 
AMPLIFIED BLUE 



AMPLIFIED RED+ 
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AMPLIFIED BLUE 



AMPLIFIED RED+ 
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INPUT (VOLTS, NORMALIZED UNITS) 



r/G. /z 



F/G. rsA 



F/G. 7S 



READ 2x2 PIXEL BLOCK: 
RG or GR or GB or BG 
GB BG RG GR 




AMPLIFY VALUE VIA 
FIRST RED GAIN AMPLIFIER 

AMPLIFY VALUE V!A 
SECOND RED GAIN AMPLiFIER 

AMPLIFY VALUE VIA 
THIRD RED GAIN AMPLiFIER 



AVERAGE TWO GREEN PIXELS 
VIA SUMMING AMPLIFIER: 
(G1 + G2)/2 




424 



426 




AMPLIFY VALUE VIA 
FIRST BLUE GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
SECOND SLUE GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
THIRD BLUE GAIN AMPLIFIER 



42S 



4J2 




440 



447 



444 



-432 



ADVANCE TO NEXT COLUMN: 
C0LUMN=C0LUMN+1 



-434 



EXCEEDING LAST RED 
PIXEL IN A ROW 

9 



'436 



ADVANCE TO NEXT ROW: 
R0W=R0W+1 



-433 



EXCEEDING LAST ROW 
JVITH RED IN A FRAME^ 



.NO 



F/G. 




\ 

AMPLIFY VALUE VIA 
FiRST GREEN GAIN AMPLIFIER 




AMPLIFY VALUE VIA 
SECOND GREEN GAIN AMPUFIE 

AMPLIFY VALUE VIA 
THIRD GREEN GAIN AMPLIFIER 






AMPLIFIED RED4 
AMPLIFIED GREEN + 
AMPLIFIED BLUE 



AMPLIFIED RED+ 
AMPLIFIED GREEN+ 
AMPLIFIED BLUE 



AMPLIFIED RED + 
AMPLIFIED GREEN + 
AMPLIFIED BLUE 



I 



-M2 



-745 



LINE SYNC 



^^f PIXEL CLOCK 



COLUMN READOUT 
SHIFT REGISTER 



DARK 
ROW 
READOUT 
CONTROL 

782-^ 
o ) 



SUBTRACTION 
BY SUMMING 

VIDEO OUT 



>- 

UJ 

< 




>- 
CO 



DARK JH> 
ROW ^04- 

EXPOSED 
ROWS 




504' 



-XL 



G- 



B 



^796 



G- 



B 



504- 



R 



R- 



G-l 



R- 



- G 



R - 



G- 



B 



G- 



B-" 



PROGRAM 

GAIN 
AMPLIFIER 



READ 2 PIXELS IN CURRENT 
COLUMN, THE DARK PIXEL AND 
THE EXPOSED PIXEL; 
D or D or D 
, RGB 



'522 



SUBTRACT DARK VALUE FROM 
EXPOSED PIXEL VALUE: 
R-D or G-D or B-D 



-524 



CLIP TO ZERO IF NEGATIVE 



nv^ 

^526 



ADVANCE TO NEXT COLUMN; 
C0LUMN = C0LUMN + 1 




523 



5JO 



ADVANCE TO NEXT ROW; 
R0W=R0W+1 



r/O, 27 




-540 




r/o. 22 



550 



READ 3 PIXELS IN CURRENT 
COLUMN, 2 DARK PIXELS AND 
THE EXPOSED PIXEL: 
D1 D1 D1 
D2 D2 D2 
RGB 



I 



-552 



AVERAGE TWO DARK PIXELS: 
AD=(D1 + D2)/2 



I 



-554 



SUBTRACT AVERAGE DARK VALUE 
FROM EXPOSED PIXEL VALUE: 
R-AD or G-AD or B-AD 




CLIP TO ZERO IF NEGATIVE 




556 



558 



ADVANCE TO NEXT COLUMN: 
C0LUMN = C0LUMN+1 





ROW SHIFT REGISTER 



I 



< 



FRAME /SYNC 



LINE SYNC 
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6JO 



'read 1x2 pixel block 



CHECK PIXEL COORDINATE 
lAGAINST THE BAD PIXEL 





6J6 



t'LACE ORIGINAL MIDDLE PIXELX 
WITH THE PREVIOUS ONE J 



ADVANCE TO NEXT COLUMN: 
COLUMN = COLUMN+I 




542 



ADVANCE TO NEXT ROW: 
ROW = ROW+1 



F/a 23 




644 



546 
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Attorney's Docket No. YMEDIA.005A 



DECLARATION - USA PATENT APPLICATION 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name; 

I believe I am the original, first and sole inventor of the subject matter which is claimed 
and for which a patent is sought on the invention entitled METHOD AND APPARATUS FOR 
CONTROLLING PIXEL SENSOR ELEMENTS; the specification of which is attached hereto; 

I hereby state that I have reviewed and understand the contents of the above identified 
specification, including the claims, as amended by any amendment referred to above; 

I acknowledge the duty to disclose information which is material to patentability as 
defined in Title 37, Code of Federal Regulations, § L56; 

I hereby claim the benefit under Title 35, United States Codes § 119(e) of any United 
States provisional application(s) listed below. 

Application No.: 60/149,796 Filing Date: August 19, 1999 

I hereby declare that all statements made herein of my own knowledge are true and that 
all statements made on information and belief are believed to be true; and fiirther that these 
statements were made with the knowledge that willfiil false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States 
Code and that such willfiil, false statements may jeopardize the validity of the appUcation or any 
patent issued thereon. 



Full name of sole inventor: Sarit Neter 

Inventor's signature .^-^^<r-^<^ ^e/^^ 

Date "XjX^^oa 

Residence: 15 Highland View, Irvine, CA 92612 

Citizenship: Israeli 

Post Office Address: same as above 



Send Correspondence To: 

KNOBBE, MARTENS, OLSON & BEAR, LLP 

Customer No. 20,995 

H:\DOCSVACC\ACC-1473.DOC:smn 
012100 



YMEDIA.005A 



PATENT 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

Applicant : Sarit Neter ) 

App, No. : Unknown ) 

Filed : Herewith ) 

For : METHOD AND APPARATUS FOR ) 

CONTROLLING PIXEL SENSOR ) 

ELEMENTS ) 

Examiner ; Unknown ) 



ESTABLISHMENT OF RIGHT OF ASSIGNEE TO TAKE ACTION 

AND 

REVOCATION AND POWER OF ATTORNEY 

Assistant Commissioner for Patents 
Washington, D.C. 20231 

Dear Sir: 

The undersigned is empowered to act on behalf of the assignee below (the "Assignee"), 
A true copy of the original Assignment of the above-captioned apphcation jfrom the inventor(s) 
to the Assignee is attached hereto. This Assignment represents the entire chain of title of this 
invention from the Inventor to the Assignee. 

I declare that all statements made herein are true, and that all statements made upon 
information and belief are believed to be true, and further, that these statements were made with 
the knowledge that willful, false statements and the like so made are punishable by fine or 
imprisonment, or both, under 18 U.S.C. § 1001, and that willfixl, false statements may jeopardize 
the vahdity of the application, or any patent issuing thereon. 

The xmdersigned hereby revokes any previous powers of attomey in the subject 
apphcation, and hereby appoints the registrants of Knobbe, Martens, Olson & Bear, LLP, 620 
Newport Center Drive, Sixteenth Floor, Newport Beach, CaUfomia 92660, Telephone 
(949) 760-0404, Customer No. 20,995, as its attomeys with full power of substitution and 



App. No. : Unknown 
Filed : Herewith 

revocation to prosecute this apphcation and to transact all business in the U.S. Patent and 
Trademark Office connected herewith. This appointment is to be to the exclusion of the inventor 
and his attorneys in accordance with the provisions of 37 C.F.R. § 3.71. 
Please use Customer No, 20,995 for all communications. 



Dated: 2^2 OO 




Ian Olsen 

Title: President & CEO 

Address: 23172 Plaza Pointe Drive, Suite 285 
LagunaHills,CA 92653 
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